Skip to content

feat(observability): add a prediction-calibration-drift alert rule for the miner - #5557

Merged
JSONbored merged 2 commits into
JSONbored:mainfrom
milosde111:feat/miner-prediction-calibration-drift-alert
Jul 13, 2026
Merged

feat(observability): add a prediction-calibration-drift alert rule for the miner#5557
JSONbored merged 2 commits into
JSONbored:mainfrom
milosde111:feat/miner-prediction-calibration-drift-alert

Conversation

@milosde111

@milosde111 milosde111 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds a gittensory-miner-prediction group to prometheus/rules/alerts.yml with a single
    rule, GittensoryMinerPredictionCalibrationDrift, that fires when the miner's predicted-gate
    accuracy drifts out of tolerance. Built against the metric shape shipped by Wire the existing miner-prediction Prometheus renderer into a real command #4838/Build the missing calibration-report join #4849:
    renderMinerPredictionMetrics (packages/gittensory-engine/src/miner-prediction-metrics.ts)
    emits gittensory_miner_prediction_correct_total / _incorrect_total, and the rule uses the
    fraction of resolved predictions that came back incorrect.
  • Mirrors the existing LoopoverHighJobFailureRatio rule's bad/(bad+good) ratio shape exactly,
    including the trailing > 0 denominator guard so a 0/0 (no resolved predictions, or an absent
    series) degrades to silent instead of false-firing — the rule ships dormant and valid until
    an operator configures a miner scrape target, the same way loopover-d1-storage stays silent
    without the D1 probe.
  • Threshold (0.5) and sustain window (for: 30m) are documented inline as the tunable knobs,
    following this file's established convention (there are no recording rules to hang a named
    constant on); a wide 6h rate window smooths the miner's naturally sparse resolution cadence.

Closes #5188.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves — Closes Add a Prometheus alert rule for prediction-calibration drift #5188.

Validation

  • git diff --check
  • npm run selfhost:validate-observability — dashboards and alert rules valid
  • npm run docs:drift-check
  • test/unit/alerts-miner-prediction-calibration-drift.test.ts (new) + the existing
    alerts-job-failure-ratio-formula, docs-selfhost-troubleshooting-metric-names, and
    observability-ci suites all pass
  • New behavior has tests: the new fixture pins the rule's formula, threshold, for/severity,
    the real metric names (tied to the engine's exported constants so a rename fails the test),
    the 0/0 silent-degradation invariant, and that it references no loopover_* server metric

If any required check was skipped, explain why:

  • This is a Prometheus alert-rule (config) change plus a test fixture. It touches no src/** code,
    so codecov/patch has nothing to instrument; no OpenAPI/MCP/UI/worker/migration surface changes,
    so those generators/gates are N/A.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. — N/A: no auth/session/network surface changes.
  • API/OpenAPI/MCP behavior is updated and tested where needed. — N/A: no API/OpenAPI/MCP surface change.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. — N/A: no UI change.
  • Visible UI changes include a UI Evidence section. — N/A: alert-rule YAML + test only, no visible surface.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs. — the alerts.yml header note was updated to stay accurate; no changelog edit.

UI Evidence

N/A — this is an observability-config (prometheus/rules/alerts.yml) + test change with no visible UI, frontend, docs page, or extension surface.

Notes

  • The runbook references only real commands (gittensory-miner calibration) and the real
    emitted counters, so an operator can act on the alert without reading the YAML.
  • Migration of any remaining hand-authored alerts onto recording-rule-based thresholds is out of
    scope; this rule follows the file's current inline-documented-threshold convention.

@milosde111
feat(prometheus): add gittensory miner prediction calibration drift a…
2b3861e 
…lert (JSONbored#5188)

This commit introduces a new alert rule for monitoring the calibration drift of the gittensory miner's predictions. The alert, named GittensoryMinerPredictionCalibrationDrift, evaluates the ratio of incorrect to total resolved predictions over a 6-hour window, ensuring it remains dormant until a miner scrape target exists. Additionally, unit tests have been added to validate the alert's configuration and behavior, ensuring it references the correct metric names and maintains proper functionality.
@milosde111
milosde111 requested a review from JSONbored as a code owner July 13, 2026 05:04
@superagent-security

superagent-security Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.91%. Comparing base (a5bec8a) to head (304a2fd).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files 
@@           Coverage Diff           @@
##             main    #5557   +/-   ##
=======================================
  Coverage   94.91%   94.91%           
=======================================
  Files         570      570           
  Lines       45351    45351           
  Branches    14675    14675           
=======================================
  Hits        43046    43046           
  Misses       1571     1571           
  Partials      734      734
Flag Coverage Δ
shard-1 44.14% <ø> (-0.01%) ⬇️
shard-2 35.73% <ø> (+0.02%) ⬆️
shard-3 31.94% <ø> (-0.15%) ⬇️
shard-4 31.85% <ø> (+0.71%) ⬆️
shard-5 32.77% <ø> (-0.10%) ⬇️
shard-6 44.34% <ø> (+0.19%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 13, 2026
@loopover-orb

loopover-orb Bot commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Tip

🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩🟩

✅ Gittensory review result - approve/merge recommended

Review updated: 2026-07-13 05:12:54 UTC

2 files · 2 AI reviewers · no blockers · readiness 100/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This PR adds a single, self-contained Prometheus alert rule (`GittensoryMinerPredictionCalibrationDrift`) mirroring the existing `LoopoverHighJobFailureRatio` bad/(bad+good) ratio shape, plus a config-fixture test that pins the rule's group, metric names, formula, and guard against the engine's exported metric-name constants. The `(sum(rate(incorrect[6h])) / (sum(rate(correct[6h])) + sum(rate(incorrect[6h])) > 0)) > 0.5` expression correctly uses the standard PromQL idiom where `X > 0` (without `bool`) filters the denominator vector to nonzero values and returns the raw sum, not a 0/1 flag, so the 0/0 guard behaves as documented (dormant, not false-firing). The change closes the linked #5188 and is narrowly scoped to the alert + its test.

Nits — 5 non-blocking
  • prometheus/rules/alerts.yml: threshold (0.5) and window (6h/30m) are only validated by convention/comment, not by any operator-facing doc cross-reference — consider linking the runbook to a troubleshooting doc entry if one exists elsewhere in the repo.
  • test/unit/alerts-miner-prediction-calibration-drift.test.ts: the regex-based assertions on `flat` are fairly brittle to formatting changes in the YAML (e.g. re-flowing the expr would break the test even if semantics are unchanged) — a parsed/normalized comparison would be more robust.
  • test/unit/alerts-miner-prediction-calibration-drift.test.ts: no test actually exercises PromQL evaluation (e.g. via promtool or a mock evaluator) to confirm the 0/0 guard behavior described in comments; it only asserts string shape, which is acceptable given the file/tooling constraints but worth flagging as a documentation-level guarantee rather than an evaluated one.
  • Consider adding a one-line pointer in the alert's runbook to any existing doc describing how an operator wires `renderMinerPredictionMetrics` output into a scrape target, since the rule is dormant until that's configured.
  • If `promtool` ever becomes available in CI, replace the regex-based expr assertions with an actual `promtool test rules` fixture for stronger guarantees.
Signal Result Evidence
Code review ✅ No blockers 2 reviewers, synthesized
Linked issue ✅ Linked #5188
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 235 registered-repo PR(s), 162 merged, 4 issue(s).
Contributor context ✅ Confirmed Gittensor contributor milosde111; Gittensor profile; 235 PR(s), 4 issue(s).
Gate result ✅ Passing No configured blocker found.
Improvement ⚠️ ℹ️ Insufficient signal risk: clean · value: insufficient-signal — Nothing measurable for the structural-improvement analyzers on this PR (e.g. no code files changed). LLM value judgment: moderate — The PR is a narrow, well-tested addition of one alert rule that closes a linked issue and follows an established in-repo pattern, but it's a small, mechanical observability addition rather than a structural improvement.
Linked issue satisfaction

Partially addressed
The PR adds a new gittensory-miner-prediction rule group with correct/incorrect metric names, for:/severity/annotations, and a 0/0-guarded ratio expr, but the drift threshold remains a bare inline `0.5` magic number rather than the named constant or recording-rule pattern the issue explicitly required, and the accompanying test only asserts the expr string via regex rather than actually evaluating

Review context
  • Author: milosde111
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: JavaScript, TypeScript, Cuda, HTML, MDX, Python, Rust
  • Official Gittensor activity: 235 PR(s), 4 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @gittensory ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @gittensory chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

  • Re-run Gittensory review

@loopover-orb loopover-orb Bot added the manual-review Gittensor contributor context label Jul 13, 2026
@JSONbored
JSONbored merged commit 5d50449 into JSONbored:main Jul 13, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JSONbored JSONbored JSONbored approved these changes

Assignees

@milosde111 milosde111

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. manual-review Gittensor contributor context

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add a Prometheus alert rule for prediction-calibration drift

2 participants

@milosde111 @JSONbored